A Visual Exploration and Statistical Analysis of a Diabetes Dataset using Python¶
By: Noureddin Sadawi, PhD¶
This Dataset is Freely Available¶
Overview:¶
The data was collected and made available by the "National Institute of Diabetes and Digestive and Kidney Diseases" as part of the Pima Indians Diabetes Database.
Diabetes.csv is available from Kaggle. We have several questions - what information is more correlated with a positive diagnosis, and if we can only ask two questions to a patient, what should we ask and how would we give them a risk of being diagnosed.
This is a machine learning database, and normally we'd just extract features, feed to a ML algorithm and sit back and relax. But we'll get our hands dirty so that you can learn more.
We'll be using Python and some of its popular data science related packages. First of all, we will import pandas to read our data from a CSV file and manipulate it for further use.
We will also use numpy to convert out data into a format suitable to feed our classification model.
We'll use seaborn and matplotlib for visualizations. We will then import Logistic Regression algorithm from sklearn. This algorithm will help us build a classification model.
++++++++++++++++++++++++++++++++++++
The following features have been provided to help us predict whether a person is diabetic or not:
- Pregnancies: Number of times pregnant
- Glucose: Plasma glucose concentration over 2 hours in an oral glucose tolerance test
- BloodPressure: Diastolic blood pressure (mm Hg)
- SkinThickness: Triceps skin fold thickness (mm)
- Insulin: 2-Hour serum insulin (mu U/ml)
- BMI: Body mass index (weight in kg/(height in m)2)
- DiabetesPedigreeFunction: Diabetes pedigree function (a function which scores likelihood of diabetes based on family history)
- Age: Age (years)
- Outcome: Class variable (0 if non-diabetic, 1 if diabetic)
Scenario: Imagine you have collected this data and wish to analyse it¶
Import required libraries¶
Load the data and view first few rows¶
1- Basic Exploration and Cleaning up¶
Missing and Zero Values¶
- It is clear the data has missing and zero values
- For example, we can see that SkinThickness = 0 in the third row
- And we can also see some 'NaN' values
- Sometimes missing values are represented by a '?'
- Let's get more info
Viewing more details about the data¶
We can use the describe function and it will give us more details about the numeric columns
The Mean and Median¶
- The mean is the simple mathematical average of a list of two or more numbers.
- The median is the middle number in a sorted, ascending or descending, list of numbers and can be more descriptive of that data set than the average.
Zero Values that Don't Make Sense¶
Some Summary Information¶
Code to display table borders¶
Image Source: 2- Useful and Informative Plots¶
Histogram Plots¶
Scatter Matrix¶
- This one is a useful one liner ... but note that it only works with numeric data
- If you want to include categorical data in there you should convert the categories into numeric labels
The scatter plot gives us both the histograms for the distributions along the diagonal, and also a lot of 2D scatter plots off-diagonal. Not that this is a symmetric matrix, so we normally just look at the diagonal and below/above it. We can see that some variables have a lot of scatter and some are correlated (ie there is a direction in their scatter). Which leads us to...
Correlation Plots¶
To easily quantify which variables / attributes are correlated with others!
https://www.statisticshowto.com/probability-and-statistics/correlation-analysis/
- Correlation is usually an indicative of good information content
- So in analysis you might want to use variables that are correlated with your outcome variable!
And you can see this is a symmetric matrix too. But it immedietly allows us to point out the most correlated and anti-correlated attributes. Some might just be common sense - Pregnancies v Age for example - but some might give us real insight into the data.
- It allows to see what we need to investigate and which variables we can focus on if we do not have the time and resources to investigate every possible column
Covariance Plot¶
Covariance is a measure of how much two random variables vary together.
In other words, it is the measure of the joint spread between two random variables.
Box Plots, Violin Plots and Bee Swarm Plots¶
2D Histograms¶
Useful when you have a lot of data ... i.e. at least 1000's of points
Contour plots¶
A bit hard to get information from the 2D histogram isnt it? Too much noise in the image. What if we try and contour diagram? We'll have to bin the data ourself. The contour API is here
- Contour plots help us make the correlation in the data a bit more obvious
- But remember .. they require a lot of data
- Looks like its just as noisy with the contour plot
- In general, for 2D histograms and contour plots, have a lot of data
- We simply don't have enough data to get smooth results!
KDE Plots¶
With these plots we smooth the data ourselves. Seaborn to the rescue!
- KDE smooths the data for you and gives you an approximation
- The smooth surface is not the real data, it is something we have done to it using a kernel density
- A kernel specifies how we are smoothing the data
- By default here we are using a Gaussian kernel
- It takes the 2D histogram and convolves it with a kernel (in the shape of a Gaussian) to smear it
No Harm in keeping it Simple¶
A scatter plot is normally fairly informative and very fast to plot.
- Looking at the scatter plot above, can we make some useful informed guesses?
- Given the level of Glucose and BMI of a random person, can we guess if they are diabetic?
Treating Points with Probability¶
Using the library ChainConsumer (examples here). written by Samuel Hinton.
It is easy to install:
pip install chainconsumer
- It gives you two contours for each chain, one for the 68% confidence interval and the other is for the 95% confidence interval
- This is useful when doing Hypothesis testing
- For example, if you randomly pick a Diabetic person out of a different data sample, you can say that 68% of the time we would expect their BMI and Glucose levels to lie within the 68% contour, and 95% of the time their BMI and Glucose levels would lie in the second contour.
- This is useful when you would like to check if a data point comes from this distrbution or not
- For example, you can check where the data point lies and estimate the chance of it being of a Diabetic or Non Diabetic person
- If we ignore the correlations, we can visually see that Diabetic individuals have higher Glucose levels that Non Diabetic individuals
- They also tend to have a higher BMI (look at the tail of the curve)
3- A Probabilistic Analysis¶
Based on the previous Correlation plot, a simple approach might be just to use the top correlated variables and investigate them further. In our case, they're: Glucose, BMI and Age.
- We will not use "Pregnancies" because we will not pay attention to Gender
- And we will not use "DiabetesPedigreeFunction" because we do not expect patients to know its value when they come to a clinic!
So its not perfect, but we can probably do an alright job approximating both these distributions as Gaussians.
Multivariate Normal Distribution¶
Allows us to model Glucose, BMI and Age as a normal in 3 dimensions rather than 3 independent normals. That way we can have correlations between them.
Plot the data in 3D and Add a Test Point¶
Very Important to Notice:¶
- You should notice how the probability is weighted.
- The weighting is done to account for the imbalance (remember there are far more patients without diabetes than with)
- We can only directly compare the two distributions if they have equal probability all up (same number of people with and without)
- This might seem odd even if the test patient is right on the maximum of our model for the diabetes patients
- This is rarely the case, and so we have to weight them
4- Predicting Diabetes with Logistic Regression¶
- Logistic Regression measures the relationship between the dependent variable (our Outcome or what we want to predict) and one or more independent variables (our features or input variables), by estimating probabilities using its underlying logistic function
- These probabilities must then be transformed into binary values in order to actually make a prediction
- This is the task of the logistic function, also called the sigmoid function
- The Sigmoid-Function is an S-shaped curve that can take any real-valued number and map it into a value between the range of 0 and 1, but never exactly at those limits
- The values between 0 and 1 will then be transformed into either 0 or 1 using a threshold classifier
More here: https://machinelearning-blog.com/2018/04/23/logistic-regression-101/
Interpreting the Model¶
To get a better sense of what is going on inside the logistic regression model, we can visualize how our model uses the different features and which features have greater effect on the outcome
Conclusions from the above figure:¶
- BMI level has the most significant influence on the model (even more than Glucose) .. does this make sense?
- The second highest influencer is age
- In third place somes the Glucose level
- The three of them have a positive influence on the prediction, i.e. their higher values are correlated with a person being diabetic
- Correlation tells us that Glucose is more correlated than BMI to the Outcome, the model relies more on BMI. This can happen for several reasons, including the fact that the correlation captured by Glucose is also captured by some other variables, whereas the information captured by BMI is not captured by other variables.
Another look at Correlation¶
Predict the same test point from above¶
Notice:¶
- Our logistic regression model agrees with our statistical model (the multivariate normal)
- Good stuff!